A hybrid (variational/Kalman) ensemble smoother for the estimation of nonlinear high-dimensional discretizations of PDE systems

Two classes of state estimation schemes, variational (4DVar) and ensemble Kalman (EnKF), have been developed and used extensively by the weather forecasting community as tractable alternatives to the standard matrix-based Kalman update equations for the estimation of high-dimensional nonlinear systems with possibly nongaussian PDFs. Variational schemes iteratively minimize a finite-horizon cost function with respect to the state estimate, using efficient vector-based gradient descent methods, but fail to capture the moments of the PDF of this estimate. Ensemble Kalman methods represent well the principal moments of the PDF, accounting for the measurements with a sequence of Kalman-like updates with the covariance of the PDF approximated via the ensemble, but fail to provide a mechanism to reinterpret past measurements in light of new data. In this paper, we first introduce a tractable method for updating an ensemble of estimates in a variational fashion, capturing correctly both the estimate (via the ensemble mean) and the leading moments of its PDF (via the ensemble distribution). We then extend this variational ensemble framework to facilitate its consistent hybridization with the ensemble Kalman smoother. Finally, it is shown (on a low-dimensional model problem) that the resulting Hybrid (variational/Kalman) Ensemble Smoother (HEnS), which inherits the tractable extensibility to high-dimensional systems of the component methods upon which it is based, significantly outperforms the existing 4DVar and EnKF approaches used operationally today for high-dimensional state estimation.